PMM-12641 Fix the dashboard upgrade #2926
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This fixes a bug where, due to a wrong ansible formula, PMM would upgrade the dashboards no matter what :)
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## v3 #2926 +/- ##
=======================================
Coverage 43.52% 43.52%
=======================================
Files 362 362
Lines 42423 42423
=======================================
Hits 18464 18464
- Misses 22386 22387 +1
+ Partials 1573 1572 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
ademidoff
requested review from
idoqo,
JiriCtvrtka and
BupycHuk
and removed request for
a team
|
Do we have any evidence that this problem existed? |
We do. To assert that, one can spin up an instance using this pipeline and run I attached the logs to the PR description. |
BupycHuk
approved these changes
Mar 25, 2024
talhabinrizwan
approved these changes
Mar 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes a bug where, due to a wrong ansible formula, PMM would upgrade the dashboards no matter what :)
PMM-12641
Link to the Feature Build: SUBMODULES-3599
Faulty logs
[ec2-user@ip-10-178-1-214 ~]$ docker exec pmm-server cat /srv/logs/pmm-update-perform-init.log ProjectName: pmm-update Version: 3.0.0 PMMVersion: 3.0.0 Timestamp: 2024-03-21 00:22:05 (UTC) FullCommit: f9fd3a7 Starting "ansible-playbook --flush-cache /opt/ansible/pmm-docker/init.yml" ... [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
PLAY [localhost] ***************************************************************
TASK [Gathering Facts] *********************************************************
ok: [localhost]
TASK [initialization : Get current version] ************************************
ok: [localhost]
TASK [initialization : Get image version] **************************************
ok: [localhost]
TASK [initialization : Set current version if VERSION doesn't exist] ***********
skipping: [localhost]
TASK [initialization : Setting current PMM version] ****************************
ok: [localhost]
TASK [initialization : Setting current PMM image version] **********************
ok: [localhost]
TASK [initialization : Set need_upgrade fact] **********************************
ok: [localhost]
TASK [initialization : Print current PMM and image versions] *******************
ok: [localhost] => {
"msg": "Current version: 3.0.0 Image Version: 3.0.0"
}
TASK [initialization : Enable maintenance mode before upgrade] *****************
ok: [localhost]
TASK [Upgrade dashboards] ******************************************************
TASK [dashboards : Get plugin list] ********************************************
ok: [localhost]
TASK [dashboards : Delete older plugins] ***************************************
changed: [localhost] => (item={'path': '/usr/share/percona-dashboards/panels/grafana-clickhouse-datasource', 'mode': '0755', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1000, 'gid': 1000, 'size': 230, 'inode': 38952651, 'dev': 39, 'nlink': 4, 'atime': 0.0, 'mtime': 1711028062.0, 'ctime': 1711222397.6157818, 'gr_name': 'pmm', 'pw_name': 'pmm', 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': True, 'xgrp': True, 'woth': False, 'roth': True, 'xoth': True, 'isuid': False, 'isgid': False})
changed: [localhost] => (item={'path': '/usr/share/percona-dashboards/panels/grafana-piechart-panel', 'mode': '0755', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1000, 'gid': 1000, 'size': 277, 'inode': 50424402, 'dev': 39, 'nlink': 4, 'atime': 0.0, 'mtime': 1711028062.0, 'ctime': 1711222397.6157818, 'gr_name': 'pmm', 'pw_name': 'pmm', 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': True, 'xgrp': True, 'woth': False, 'roth': True, 'xoth': True, 'isuid': False, 'isgid': False})
changed: [localhost] => (item={'path': '/usr/share/percona-dashboards/panels/grafana-polystat-panel', 'mode': '0755', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1000, 'gid': 1000, 'size': 255, 'inode': 1117878, 'dev': 39, 'nlink': 5, 'atime': 0.0, 'mtime': 1711028062.0, 'ctime': 1711222397.6157818, 'gr_name': 'pmm', 'pw_name': 'pmm', 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': True, 'xgrp': True, 'woth': False, 'roth': True, 'xoth': True, 'isuid': False, 'isgid': False})
changed: [localhost] => (item={'path': '/usr/share/percona-dashboards/panels/grafana-worldmap-panel', 'mode': '0755', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1000, 'gid': 1000, 'size': 188, 'inode': 17361044, 'dev': 39, 'nlink': 6, 'atime': 0.0, 'mtime': 1711028062.0, 'ctime': 1711222397.6157818, 'gr_name': 'pmm', 'pw_name': 'pmm', 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': True, 'xgrp': True, 'woth': False, 'roth': True, 'xoth': True, 'isuid': False, 'isgid': False})
changed: [localhost] => (item={'path': '/usr/share/percona-dashboards/panels/jdbranham-diagram-panel', 'mode': '0755', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1000, 'gid': 1000, 'size': 175, 'inode': 38952661, 'dev': 39, 'nlink': 3, 'atime': 0.0, 'mtime': 1711028062.0, 'ctime': 1711222397.6157818, 'gr_name': 'pmm', 'pw_name': 'pmm', 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': True, 'xgrp': True, 'woth': False, 'roth': True, 'xoth': True, 'isuid': False, 'isgid': False})
changed: [localhost] => (item={'path': '/usr/share/percona-dashboards/panels/natel-discrete-panel', 'mode': '0755', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1000, 'gid': 1000, 'size': 191, 'inode': 46595502, 'dev': 39, 'nlink': 4, 'atime': 0.0, 'mtime': 1711028062.0, 'ctime': 1711222397.6157818, 'gr_name': 'pmm', 'pw_name': 'pmm', 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': True, 'xgrp': True, 'woth': False, 'roth': True, 'xoth': True, 'isuid': False, 'isgid': False})
changed: [localhost] => (item={'path': '/usr/share/percona-dashboards/panels/petrslavotinek-carpetplot-panel', 'mode': '0755', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1000, 'gid': 1000, 'size': 4096, 'inode': 58998806, 'dev': 39, 'nlink': 7, 'atime': 0.0, 'mtime': 1711028062.0, 'ctime': 1711222397.6157818, 'gr_name': 'pmm', 'pw_name': 'pmm', 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': True, 'xgrp': True, 'woth': False, 'roth': True, 'xoth': True, 'isuid': False, 'isgid': False})
changed: [localhost] => (item={'path': '/usr/share/percona-dashboards/panels/pmm-app', 'mode': '0755', 'isdir': True, 'ischr': False, 'isblk': False, 'isreg': False, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 1000, 'gid': 1000, 'size': 18, 'inode': 42566029, 'dev': 39, 'nlink': 3, 'atime': 0.0, 'mtime': 1711028062.0, 'ctime': 1711222397.6157818, 'gr_name': 'pmm', 'pw_name': 'pmm', 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': True, 'xgrp': True, 'woth': False, 'roth': True, 'xoth': True, 'isuid': False, 'isgid': False})
TASK [dashboards : Copy plugins to the plugin directory] ***********************
changed: [localhost]
TASK [dashboards : Set permissions for the plugin directory] *******************
changed: [localhost]
TASK [dashboards : Restart grafana with new plugins] ***************************
changed: [localhost] => (item=stop)
changed: [localhost] => (item=remove)
changed: [localhost] => (item=add)
TASK [initialization : Copy file with image version] ***************************
ok: [localhost]
TASK [initialization : Create a backup directory] ******************************
ok: [localhost]
TASK [initialization : Output pmm-managed logs] ********************************
changed: [localhost]
TASK [initialization : Wait for PMM to be ready] *******************************
ok: [localhost]
TASK [initialization : Disable maintenance mode] *******************************
changed: [localhost]
PLAY RECAP *********************************************************************
localhost : ok=18 changed=6 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
The fix has been tested on the feature build instance. ✅
Logs (cat /srv/logs/pmm-update-perform-init.log)
Starting "ansible-playbook --flush-cache /opt/ansible/pmm-docker/init.yml" ... [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
PLAY [localhost] ***************************************************************
TASK [Gathering Facts] *********************************************************
ok: [localhost]
TASK [initialization : Get current version] ************************************
ok: [localhost]
TASK [initialization : Get image version] **************************************
ok: [localhost]
TASK [initialization : Set current version if VERSION doesn't exist] ***********
skipping: [localhost]
TASK [initialization : Setting current PMM version] ****************************
ok: [localhost]
TASK [initialization : Setting current PMM image version] **********************
ok: [localhost]
TASK [initialization : Set need_upgrade fact] **********************************
ok: [localhost]
TASK [initialization : Print current PMM and image versions] *******************
ok: [localhost] => {
"msg": "Current version: 3.0.0 Image Version: 3.0.0"
}
TASK [initialization : Print need_upgrade fact] ********************************
ok: [localhost] => {
"msg": "Need upgrade: False"
}
TASK [initialization : Enable maintenance mode before upgrade] *****************
skipping: [localhost]
TASK [Upgrade dashboards] ******************************************************
skipping: [localhost]
TASK [initialization : Copy file with image version] ***************************
skipping: [localhost]
TASK [initialization : Create a backup directory] ******************************
skipping: [localhost]
TASK [initialization : Wait for PMM to be ready] *******************************
skipping: [localhost]
TASK [initialization : Disable maintenance mode] *******************************
skipping: [localhost]
TASK [initialization : Output pmm-managed logs] ********************************
changed: [localhost]
PLAY RECAP *********************************************************************
localhost : ok=9 changed=1 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0
NOTE: PMM v2 is not affected.